United  States  Government  Accountability  Office 


GAO 


Report  to  Congressional  Committees 


SMALL  BUSINESS 

INNOVATION 

RESEARCH 

Agencies  Need  to 
Strengthen  Efforts  to 
Improve  the 
Completeness, 
Consistency,  and 
Accuracy  of  Awards 
Data 


GAO 

Accountability  *  Integrity  *  Reliability 


GAO-07-38 


Report  Documentation  Page 

Form  Approved 

0MB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 

1.  REPORT  DATE 

OCT  2006 

2.  REPORT  TYPE 

3.  DATES  COVERED 

00-10-2006  to  00-10-2006 

4.  TITLE  AND  SUBTITLE 

5a.  CONTRACT  NUMBER 

Small  Business  Innovation  Research.  Agencies  Need  to  Strengthen  Efforts 
to  Improve  the  Completeness,  Consistency,  and  Accuracy  of  Awards  Data 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

U.S.  Government  Accountability  Office, 441  G  Street 

NW, Washington, DC, 20548 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

The  original  document  contains  color  images. 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

18.  NUMBER 
OF  PAGES 

36 

19a.  NAME  OF 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


October  2006 


A  GAO 

Accountability  Integrity  Reliability 

Highlights 

Highlights  of  GAO-07-38,  a  report  to 
congressional  committees 


SMALL  BUSINESS  INNOVATION 
RESEARCH 

Agencies  Need  to  Strengthen  Efforts  to 
Improve  the  Completeness,  Consistency, 
and  Accuracy  of  Awards  Data 


Why  GAO  Did  This  Study 

The  Small  Business  Innovation 
Research  (SBIR)  program  was 
created  to  increase  the  use  of  small 
businesses  to  meet  federal  research 
needs  and  commercialize  the 
results  of  this  research.  To 
monitor  the  program,  the  Small 
Business  Administration  (SBA) 
requires  participating  agencies  to 
provide,  in  a  standard  format, 
specific  data  on  all  SBIR  awards 
they  make.  SBA  then  compiles 
these  data  into  a  database  known 
as  Tech-Net.  Congress  also 
required  SBA  to  create,  by  2001,  a 
restricted  and  more  comprehensive 
database  that  would  provide 
information  for  government 
agencies  to  use  in  evaluating  the 
program.  GAO  was  asked  to 
identify  the  (1)  types  of  data  that 
agencies  report  to  SBA  for 
inclusion  in  the  Tech-Net  database, 
(2)  extent  to  which  these  data  are 
provided  in  a  standard  format,  (3) 
extent  to  which  SBA  has 
established  the  government-use 
database,  and  (4)  extent  to  which 
SBIR  agencies  have  developed  and 
implemented  techniques  to  track 
commercialization  of  SBIR 
projects.  GAO  reviewed  8  of  the  11 
agencies  participating  in  SBIR. 


What  GAO  Recommends 


GAO  recommends  that  SBA  and 
SBIR  participating  agencies  work 
together  to  improve  the  quality  of 
the  data  in  SBA’s  Tech-Net 
database.  SBA  and  the  SBIR 
participating  agencies  included  in 
this  report  generally  agreed  with 
our  recommendation. 


www.gao.gov/cgi-bin/getrpt7GAO-07-38. 

To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  the  link  above. 

For  more  information,  contact  Anu  K.  Mittal  at 
(202)512-3841  ormittala@gao.gov. 


What  GAO  Found 

Federal  agencies  participating  in  the  SBIR  program  annually  submit  over  40 
data  elements  to  SBA  for  each  award  they  make.  These  data  include 
information  on  the  award,  such  as  value  and  a  descriptive  abstract; 
information  on  the  recipient,  such  as  name  and  gender;  and  information 
about  the  firm  receiving  the  award,  such  as  number  of  employees  and 
location.  Participating  agencies  submit  most  of  the  information  required  by 
SBA,  but  they  are  not  consistently  providing  all  required  data  elements, 
including  the  number  of  employees  in  the  firm,  and  the  gender  and  socio¬ 
economic  status  of  the  award  recipient,  resulting  in  incomplete  sections  of 
the  database.  Agencies  stated  that  this  happens  because  they  do  not  collect 
all  of  the  information  that  SBA  wants  and  because  SBA’s  requirements 
change  regularly. 

Some  participating  agencies  are  not  submitting  SBIR  award  data  in  the 
standard  format  required  by  SBA,  and  although  SBA’s  quality  assmance 
processes  correct  most  of  these  problems,  they  do  not  correct  all  of  them.  In 
2004  and  2005,  about  25  percent  of  the  data  provided  by  five  participating 
agencies  did  not  comply  with  SBA’s  format.  Formatting  inconsistencies 
occm  because  the  template  SBA  has  provided  agencies  for  reporting  data 
can  be  edited.  According  to  SBA,  identifying  and  correcting  inconsistently 
formatted  data  involves  considerable  resomces,  therefore  the  agency  has 
focused  its  quality  assmance  efforts  only  on  key  data  elements  needed  to 
track  awards;  other  fields,  such  as  those  containing  demographic  data,  are 
generally  not  corrected.  As  a  result,  comprehensive  program  evaluations 
may  be  limited  by  the  quality  of  the  data  in  these  fields.  SBA  officials  expect 
this  problem  to  be  resolved  by  fiscal  year  2007,  when  all  data  will  be 
submitted  via  an  Internet  interface  that  will  not  allow  changes  in  the  format. 

SBA  is  5  years  behind  schedule  in  meeting  the  congressional  mandate  to 
implement  a  restricted  goveimnent-use  database  for  the  SBIR  program.  SBA 
had  planned  to  meet  this  requirement  by  expanding  its  Tech-Net  database  to 
include  a  restricted  government-use  section.  SBA  officials  attributed  the 
delay  in  meeting  the  2001  deadline  primarily  to  increased  secmity 
requirements  needed  for  the  database,  agency  management  changes,  and 
budgetary  constraints.  SBA  officials  expect  the  government-use  section  of 
Tech-Net  to  be  operational  by  October  1,  2006,  when  safeguards  to  protect 
the  proprietary  commercialization  information  in  the  database  are  in  place. 

Most  agencies  GAO  reviewed  systematically  gather  data  on  the 
commercialization  success  of  SBIR-funded  projects.  Five  of  these  eight 
agencies  regularly  survey  all  awardees  to  gather  information  on  program 
participation,  including  commercial  success,  and  one  agency  is  about  to  start 
a  similar  survey.  In  contrast,  two  agencies  only  gather  anecdotal  success 
stories  from  a  small  sample  of  SBIR  awardees.  SBA  and  agency  officials 
generally  agree  that  despite  their  best  efforts,  obtaining  commerciahzation 
information  from  awardees  remains  a  major  challenge. 
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United  States  Government  Acconntability  Office 
Washington,  DC  20548 


October  19,  2006 

The  Honorable  Sherwood  Boehlert 
Chairman 

The  Honorable  Bart  Gordon 
Ranking  Minority  Member 
Committee  on  Science 
House  of  Representatives 

The  Honorable  Vernon  J.  Ehlers 
Chairman 

The  Honorable  David  Wu 
Ranking  Minority  Member 
Subcommittee  on  Environment,  Technology 
and  Standards 
Committee  on  Science 
House  of  Representatives 

Recognizing  the  potential  of  small  businesses  to  be  a  source  of  significant 
innovation,  the  Congress  established  the  Small  Business  Irmovation 
Research  (SBIR)  program  in  1982.  From  fiscal  year  1983  through  fiscal 
year  2004,  federal  agencies  that  participated  in  the  SBIR  program  awarded 
over  $17  billion  in  grants,  contracts,  or  cooperative  agreements  to  over 
82,000  projects.  The  primary  goals  of  the  SBIR  program  are  to  stimulate 
technological  innovation,  meet  federal  research  and  development  (R&D) 
needs,  foster  participation  by  minority  and  disadvantaged  persons  in 
technological  innovation,  and  increase  the  commercial  success 
(commercialization)  of  innovation  that  is  derived  from  federally  funded 
R&D.^  Because  the  Congress  did  not  define  what  constitutes  commercial 
success  of  federally  funded  R&D  or  how  best  to  measure  it,  agencies  have 
used  different  commercialization  outcomes  for  the  SBIR  program,  such  as 
the  sale  of  the  resulting  SBIR-funded  product  or  process,  the  extent  to 
which  SBIR  firms  have  received  non-SBIR  funding,  or  the  creation  of  new 
jobs  or  products.  The  SBIR  program  is  currently  scheduled  to  expire  on 
September  30,  2008. 


^Small  Business  Innovation  Development  Act  of  1982,  Pub.  L.  No.  97-219  (1982). 
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Every  federal  agency  with  an  R&D  budget  of  $100  million  or  more  is 
required  to  establish  and  operate  an  SBIR  program  funded  by  2.5  percent 
of  the  agency’s  budget  for  research  conducted  by  others,  called  extramural 
research.  Currently,  11  federal  agencies  participate  in  the  SBIR  program.^ 
Each  agency  manages  its  own  program,  including  targeting  research  areas, 
reviewing  proposed  projects,  and  making  research  awards  through  grants, 
contracts,  or  cooperative  agreements.  The  Small  Business  Administration 
(SBA)  plays  a  central  administrative  role  by,  for  example,  issuing  policy 
directives  to  the  participating  federal  agencies,  collecting  data  from 
participating  agencies  on  awards  and  recipients,  and  reporting  program 
results  annually  to  the  Congress. 

Over  the  last  24  years,  the  SBIR  program  has  been  reauthorized  and 
modified  by  the  Congress  at  various  times.  For  example,  the  Small 
Business  Research  and  Development  Enhancement  Act  of  1992  directed 
SBA  and  participating  agencies  to,  among  other  things,  emphasize  the  goal 
of  increasing  commercialization  of  research  results  and  to  improve  the 
government’s  dissemination  of  program-related  data.^  As  a  result,  agencies 
were  required  to  include  commercialization  potential  as  a  criterion  for 
selecting  award  recipients.  During  this  same  period,  SBA  began  to  develop 
a  publicly  available  database,  known  as  Tech-Net,  that  contained 
information  on  all  awards  made  through  the  SBIR  program.^  The  Tech-Net 
database  is  intended  to  be,  among  other  things,  an  electronic  gateway  of 
technology  information  and  resources  for  researchers,  scientists,  and 
government  officials  about  federally-funded  leading  edge  technology 
research.  The  Small  Business  Innovation  Research  Program 
Reauthorization  Act  of  2000  formalized  this  database  by  requiring  SBA  to 
develop,  maintain,  and  make  available  to  the  public  a  searchable,  up-to- 
date,  electronic  database  that  contained  SBIR  award  information.^  The 


^The  eleven  SBIR  participating  agencies  tire  the  departments  of  Agriculture  (USD A), 
Commerce,  Defense  (DOD),  Education,  Energy  (DOE),  Health  and  Human  Services, 
Homeland  Security,  and  Transportation;  the  Environmental  Protection  Agency  (EPA);  the 
National  Aeronautics  and  Space  Administration  (NASA);  and  the  National  Science 
Foundation  (NSF). 

¥ub.  L.  No.  102-564  (1992). 

^The  Tech-Net  database  also  contains  award  information  on  SBA’s  Small  Business 
Technology  Transfer  (STTR)  program.  However,  the  focus  of  this  report  is  on  the  SBIR 
program  and  not  the  STTR  program.  As  such,  any  discussion  of  the  Tech-Net  database  only 
refers  to  the  information  in  the  database  related  to  the  SBIR  awards. 

¥ub.  L.  No.  106-564  (2000). 
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2000  reauthorization  act  also  required  SBA  to  develop  and  maintain 
another  restricted  government  database  that  would  contain  additional 
information  on  commercialization  not  contained  in  the  public  Tech-Net 
database,  thereby  allowing  better  evaluations  of  the  SBIR  program  on  an 
ongoing  basis.  This  database  was  to  be  established  by  mid-2001  and  made 
available  only  to  government  agencies  and  certain  other  authorized  users. 

As  the  program  has  evolved  over  time,  congressional  direction  has 
focused,  among  other  things,  on  the  ability  of  SBIR  award  recipients  to 
commercialize  the  results  of  their  research,  as  evidenced  by  the  increased 
emphasis  on  commercialization  in  the  1992  and  2000  reauthorizations  of 
the  program.  At  various  points  in  the  life  of  the  program,  we  have  reported 
that  the  SBIR  program  has  been  successful  in  increasing 
commercialization  of  research  results  and  that  agencies  have  used  various 
methods  to  measure  the  commercial  success  of  the  projects  they  fund.® 
Largely  these  methods  have  consisted  of  surveys  of  award  recipients  to 
obtain  data  on  indicators  of  commercial  success  and  soliciting  “success 
stories”  voluntarily  provided  by  SBIR  award  recipients.  However,  we  also 
reported  that  these  methods  provided  only  “snapshots”  of  commercial 
success  and  did  not  allow  for  a  systematic  demonstration  of  changes  in 
program  commercialization  rates  over  time.  We  reported  that  the  lack  of 
clarity  on  how  much  emphasis  agencies  should  place  on 
commercialization  versus  other  SBIR  program  goals  had  also  created 
challenges  for  assessing  the  program’s  results. 

In  the  context  of  efforts  to  monitor  and  evaluate  the  success  of  the  SBIR 
program,  you  requested  that  we  identify  the  (1)  types  of  data  that 
participating  SBIR  agencies  are  reporting  to  SBA  for  inclusion  in  the  Tech- 
Net  database;  (2)  extent  to  which  agencies  provide  data  for  the  Tech-Net 
database  in  a  standard  format  to  enable  program  evaluation;  (3)  extent  to 
which  SBA  has  met  the  mandate  to  establish,  by  mid-2001,  a  government- 
use  database  that  can  be  used  for  program  evaluation;  and  (4)  extent  to 
which  participating  SBIR  agencies  have  developed  and  implemented 
techniques  to  track  the  commercialization  success  of  SBIR  projects. 


®See  GAO,  Federal  Research:  Observations  on  the  Small  Business  Innovation  Research 
Program,  GAO-05-861T,  June  28,  2006;  Federal  Research:  Observations  on  the  Small 
Business  Innovation  Program,  GAO/T-RCED-98-218,  June  4,  1998;  Federal  Research: 
Small  Business  Innovation  Research  Shows  Success  but  Can  be  Strengthened, 
GAO/RCED-92-37,  Mar.  30,  1992;  and  Federal  Research:  Assessment  of  Small  Business 
Innovation  Research  Programs,  GAO/RCED-89-39,  Jan.  23,  1989. 
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In  conducting  this  study,  we  reviewed  the  SBIR-related  activities  at  8  of 
the  11  SBIR  participating  agencies,  which  account  for  over  98  percent  of 
the  total  dollars  awarded  by  the  program  in  fiscal  year  20057  To  determine 
the  types  of  data  these  participating  agencies  are  reporting  to  SBA  and  the 
extent  to  which  SBA  has  complied  with  the  requirement  to  establish  a 
government-use  database  that  can  be  used  for  program  evaluation 
purposes,  we  compared  the  provisions  in  the  Small  Business  Innovation 
Research  Program  Reauthorization  Act  of  2000,  SBA’s  amended  Policy 
Directive  implementing  the  Act  issued  in  September  2002,  and  other 
guidance  with  agency  SBIR  Tech-Net  database  reports.  To  determine  the 
extent  to  which  data  for  the  Tech-Net  database  are  provided  in  a  standard 
format  to  enable  program  evaluation,  we  compared  the  data  from 
participating  agencies  with  data  in  SBA’s  Tech-Net  database  for  fiscal 
years  2004  and  2005,  the  2  most  recent  years  for  which  data  were 
available.  Our  data  reliability  review  focused  on  SBA’s  data  system  and 
internal  controls,  rather  than  on  the  systems  and  internal  controls 
agencies  use  to  create  the  data  provided  to  SBA.  To  assess  the  reliability  of 
the  data  in  SBA’s  Tech-Net  database,  we  interviewed  SBA  officials  about 
the  database  and  reviewed  related  documentation.  We  determined  that  the 
data  are  sufficiently  reliable  for  our  purpose,  which  is  to  report  on  SBA’s 
efforts  to  ensure  consistency  and  completeness  of  the  data  it  receives.  We 
used  GAO’s  data  reliability  guidance  to  identify  key  attributes  of  data 
quality  that  can  facilitate  program  evaluation.  We  also  interviewed  SBA 
and  agency  officials  to  determine  the  extent  to  which  the  government-use 
database  requirements  have  been  implemented.  To  determine  the  extent  to 
which  participating  agencies  and  SBA  have  developed  and  implemented 
techniques  to  evaluate  commercialization  success  of  SBIR  projects,  we 
reviewed  agency  documentation  and  interviewed  SBA  and  agency 
officials.  Appendix  I  contains  a  detailed  discussion  of  the  scope  and 
methodology  of  our  review.  We  conducted  our  review  from  April  2006 
through  September  2006  in  accordance  with  generally  accepted 
government  auditing  standards. 


Results  in  Brief 


Federal  agencies  participating  in  the  SBIR  program  annually  submit  to 
SBA  a  wide  range  of  descriptive  information  about  each  award  they  make. 
Each  year  SBA  requires  participating  agencies  to  provide  over  40  data 


^The  SBIR  participating  agencies  included  in  this  review  are  DOD,  DOE,  ERA,  NASA, 
National  Institutes  of  Health  (NIH)  within  the  Department  of  Health  and  Human  Services, 
National  Institute  of  Standards  and  Technology  (NIST)  within  the  Department  of 
Commerce,  NSF,  and  USDA. 
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elements  for  each  SBIR  award  made.  These  data  include  information 
about  the  award  itself,  such  as  the  award  amount,  a  descriptive  abstract  of 
the  project,  and  a  unique  tracking  number;  information  about  the  award 
recipient  such  as  the  name,  gender,  and  socio-economic  status;  and 
information  about  the  type  of  firm  that  received  the  award,  such  as 
number  of  employees  and  geographic  location.  However,  we  determined 
that  although  participating  agencies  are  providing  most  of  the  information 
requested  by  SBA,  they  are  not  consistently  providing  the  full  range  of 
required  information.  For  example,  two  of  the  eight  agencies  we  reviewed 
had  not  consistently  provided  SBA  data  on  the  gender  and  socio-economic 
status  of  SBIR  award  recipients,  and  five  did  not  provide  data  on  the 
number  of  employees  working  at  the  firms  that  received  SBIR  awards.  As 
a  result,  certain  sections  of  the  database  needed  for  comprehensive 
program  evaluation  are  incomplete.  Agencies  cited  a  variety  of  reasons  for 
not  providing  the  data  requested  by  SBA,  including  frequent  changes  in 
SBA’s  data  requirements  and  differences  in  the  types  of  data  agencies 
collect  versus  the  types  of  data  that  SBA  wants  to  collect. 

Some  participating  agencies  are  not  submitting  SBIR  award  data  in  the 
standard  format  required  by  SBA,  and  while  some  of  these  problems  are 
corrected  by  SBA’s  quality  assurance  processes,  others  are  not.  Since 
2000,  SBA  officials  have  worked  with  participating  agencies  to  ensure 
greater  consistency  in  how  the  data  required  for  the  Tech-Net  database  are 
formatted  and  have  developed  a  reporting  template  that  includes  the 
required  data  fields  and  instructions  for  appropriate  data  entry.  However, 
almost  a  quarter  of  the  data  provided  by  five  participating  agencies  in  2004 
and  2005  still  did  not  comply  with  SBA’s  formatting  guidance.  As  a  result, 
SBA  officials  said  that  they  have  to  spend  a  considerable  amount  of  time 
and  resources  correcting  these  data,  and  are  unable  to  correct  all  of  them. 
SBA’s  current  quality  assurance  efforts  therefore  focus  on  obtaining 
complete  and  accurate  data  only  for  those  fields  essential  to  tracking 
specific  awards,  such  as  the  tracking  numbers  and  award  amounts,  and 
not  correcting  data  in  fields  that  contain  demographic  information  about 
award  recipients.  The  primary  reason  for  the  inconsistently  formatted  data 
is  that  the  template  SBA  has  provided  for  reporting  data  can  be  edited,  and 
some  agencies  have  done  so  and  introduced  different  formats.  According 
to  SBA,  by  fiscal  year  2007,  this  issue  should  be  resolved  because  agencies 
will  have  to  submit  their  data  through  an  Internet  interface  that  will  no 
longer  allow  the  agencies  to  change  the  format  of  titles  and  data.  In  light  of 
the  problems  we  identified  with  the  Tech-Net  database  and  the 
implications  for  these  errors  to  limit  evaluations  of  the  SBIR  program,  we 
are  recommending  that  SBA  work  with  participating  agencies  to 
strengthen  efforts  to  improve  the  quality  of  the  data. 
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SBA  has  not  met  the  congressional  mandate  to  develop  and  implement,  by 
June  2001,  a  government-use  database  that  can  be  used  for  evaluating  the 
SBIR  program.  To  meet  the  congressional  requirement  to  develop  and 
implement  a  government-use  database,  SBA  planned  to  expand  the 
existing  Tech-Net  database  and  include  a  restricted  government-use 
section  that  would  be  accessible  only  to  government  agencies  and  other 
authorized  users.  This  government-use  section  of  the  Tech-Net  database 
would  rely  on  information  already  gathered  for  the  Tech-Net  database  and 
supplemented  by  information  on  the  commercialization  outcomes  for 
SBIR  awards.  However,  SBA  officials  told  us  that  they  have  been  unable  to 
meet  the  requirement  to  implement  a  government-use  database  by  2001 
because  of  management  changes  that  have  occurred  at  the  agency  and 
because  of  budgetary  constraints.  To  date,  with  the  help  of  two 
contractors,  SBA  has  developed  the  framework  for  (1)  importing  data  into 
the  government-use  section  of  the  database,  and  (2)  an  Internet-based 
interface  that  would  allow  agencies  and  award  recipients  to  access  and 
enter  commercialization  information.  According  to  SBA,  the  government- 
use  section  of  the  database  will  not  be  fully  operational  until  the  agency 
resolves  certain  outstanding  issues,  such  as  making  the  commercialization 
information  secure  because  this  information  is  considered  proprietary  and 
confidential.  SBA  officials  expect  that  the  government-use  section  of  the 
database  will  be  operational  by  October  1,  2006;  however,  they  also 
recognize  that  additional  enhancements  to  improve  the  efficiency  of  the 
database  may  still  be  needed  after  the  system  is  operational. 

Seven  of  the  eight  agencies  participating  in  the  SBIR  program  that  we 
reviewed  are  gathering  data  on  the  commercialization  success  of  SBIR- 
funded  projects;  however,  the  methodological  rigor  of  their  methods 
varies  significantly.  Under  the  program’s  authorizing  legislation  and 
implementation  guidance,  agencies  have  been  given  considerable 
flexibility  in  tracking  the  commercial  success  of  their  SBlR-funded 
projects.  Specifically,  five  of  the  eight  agencies  systematically  and 
periodically  survey  SBIR  award  recipients  to  gather  a  range  of  information 
on  program  participation,  including  commercialization  experience.  For 
example,  DOD  annually  surveys  SBIR  award  recipients  to  gather  data  on 
(1)  sales  of  SBlR-funded  research  results,  (2)  commitments  SBIR 
recipients  have  received  for  additional  development  funding,  and  (3) 
whether  SBlR-funded  research  results  have  been  used  by  other  federal 
programs.  Two  of  the  eight  agencies  we  reviewed  use  less  systematic  data 
gathering  efforts  and  instead  focus  on  gathering  success  stories  and 
conducting  periodic  follow-up  with  a  small  sample  of  SBIR  award 
recipients.  For  example,  USDA  periodically  contacts  a  small  group  of 
award  recipients  to  update  and  obtain  information  on  their 


Page  6 


GAO-07-38  Small  Business  Innovation  Research 


commercialization  experiences.  The  remaining  agency — NIST — ^has 
developed  a  survey  that  it  plans  to  send  to  all  of  its  SBIR  award  recipients 
in  2006.  Regardless  of  the  methods  used  to  gather  commercialization 
information,  SBA  and  agency  officials  believe  that  several  factors 
complicate  their  efforts  to  obtain  this  information  in  a  standardized  and 
complete  manner.  For  example,  these  officials  told  us  that  many  years 
may  elapse  from  the  time  an  SBIR  award  is  granted  to  the  time  a  product 
or  process  achieves  commercial  success  and  maintaining  contact  with 
award  recipients  during  this  time  period  is  often  difficult.  Moreover, 
during  this  time,  firms  may  change  their  names,  be  purchased  by  other 
firms,  or  start  new  businesses  to  pursue  the  project’s  commercial 
potential,  making  it  even  more  difficult  for  the  agencies  to  track  these 
firms  and  link  them  to  their  original  SBIR  awards. 


Background 


The  Small  Business  Innovation  Development  Act  of  1982  established  the 
SBIR  program  and  identified  four  program  goals:  technological  innovation, 
commercialization  of  the  research  results,  the  use  of  small  businesses  to 
meet  agencies’  R&D  needs,  and  participation  in  federal  R&D  by  minorities 
and  disadvantaged  persons.  The  legislation  provided  for  a  competitive 
three-phased  program:  phase  I  to  determine  the  technical  and  scientific 
merit  and  feasibility  of  a  proposed  research  idea;  phase  II  to  further 
develop  the  idea,  taking  into  account  its  commercial  potential;  and  phase 
III  to  commercialize  the  resulting  product  or  process  with  private  or 
federal  investment  but  no  additional  SBIR  funding. 

Under  the  SBIR  program  provisions,  federal  agencies  that  have  external 
R&D  budgets  of  $100  million  or  more  are  required  to  use  2.5  percent  of 
these  budgets  to  establish  and  operate  an  SBIR  program.  SBA  oversees 
and  coordinates  the  efforts  of  the  eleven  agencies  currently  participating 
in  the  program.  In  this  capacity,  SBA  coordinates  the  participating 
agencies’  schedules  to  announce  opportunities  for  firms  to  apply  for 
awards,  called  a  solicitation,  and  provides  access  to  these  solicitations 
through  its  Web  site.  As  part  of  its  oversight  effort,  SBA  collects  SBIR  data 
from  the  participating  agencies,  aggregates  the  data,  and  uses  the  data  to, 
among  other  things,  monitor  the  program  and  report  annually  to  the 
Congress. 

In  reauthorizing  the  SBIR  program  in  1992,  the  Congress  stated  its 
intention  to  expand  and  improve  the  program  by  emphasizing  its  goal  of 
increasing  private  sector  commercialization,  increasing  participation  in 
federal  R&D  by  small  businesses,  and  improving  the  government’s 
dissemination  of  program-related  information.  One  of  the  new  provisions 
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under  the  1992  legislation  requires  agencies,  when  evaluating  phase  11 
proposals,  to  consider  their  commercial  potential,  including  the  recipients’ 
experiences  commercializing  the  results  from  previous  SBIR  awards, 
commitments  accompanying  the  proposals  for  developmental  funding 
from  sources  other  than  the  SBIR  program,  and  other  factors. 

The  SBIR  program  was  again  reauthorized  in  2000  by  the  Small  Business 
Irmovation  Research  Program  Reauthorization  Act  of  2000.  The  2000 
legislation  directed  SBA  and  participating  agencies  to,  among  other  things, 
expand  the  scope  of  publicly  available  information  on  specific  awards  and 
to  annually  report  data  on  their  SBIR  programs  to  SBA.  The  act  required 
that  SBIR  phase  II  award  recipients  be  requested  to  voluntarily  provide 
information  to  the  agencies  describing  the  outputs  and  outcomes  of  their 
SBIR  award.  The  act  also  required  SBA  to  establish,  by  mid-2001,  a 
searchable  and  up-to-date  electronic  database  available  for  public  use,  and 
a  restricted  government-use  database.  To  accomplish  this  mandate,  SBA 
envisioned  expanding  the  electronic  database,  known  as  Tech-Net,  that  it 
had  developed  in  the  late  1990s,  into  two  sections:  a  public-use  section  and 
a  restricted  government-use  section.  The  public-use  section  of  the  Tech- 
Net  database  would  provide  access  to  nearly  all  of  the  statutorily-required 
award  information  for  SBIR  awards  gathered  by  the  agencies.  The  public- 
use  section  was  intended  to  be  an  electronic  gateway  of  technology 
information  and  resources  for  researchers,  scientists,  and  government 
officials  who  are  seeking  information  on  potential  small  business  partners, 
contractors,  or  leading  edge  technology  research.  The  government-use 
section  would  be  solely  used  for  program  evaluation  purposes  accessible 
only  to  government  agencies  and  other  authorized  users,  and  would 
contain  commercialization  data  voluntarily  supplied  by  SBIR  recipients 
upon  completion  of  their  phase  II  SBIR  funding  agreement,  such  as 
revenue  from  the  sale  of  new  products  or  services  resulting  from  the 
research  undertaken  with  the  award.  In  addition,  applicants  for  phase  II 
awards  would  be  required  to  update  information  on  the  commercialization 
success  of  any  prior  SBIR  awards  they  had  received.  Currently,  SBA  has 
created  and  is  maintaining  the  public-use  section  of  the  Tech-Net 
database,  which  is  available  on  the  Internet  to  the  general  public. 
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Agencies  Provide 
Most  of  the  Required 
SBIR-Award  Data 
Elements  to  SBA,  but 
Some  Data  Submitted 
are  Incomplete 


While  federal  agencies  participating  in  the  SBIR  program  submit  a  wide 
range  of  descriptive  award  information  to  SBA  annually,  these  agencies 
are  not  consistently  providing  all  of  the  required  information.  As  outlined 
in  SBA’s  guidance,  each  year  SBIR  participating  agencies  are  required  to 
collect  and  maintain  information  from  recipients  and  provide  it  to  SBA  so 
that  it  can  be  included  in  the  Tech-Net  database.  Specifically,  participating 
agencies  are  required  to  provide  over  40  data  elements  for  each  SBIR 
award  they  make.  These  data  include  award-specific  information,  such  as 
the  date  and  amount  of  the  award,  an  abstract  of  the  project  funded  by  the 
award,  and  a  unique  tracking  number  for  each  award.  Participating 
agencies  are  also  required  to  provide  data  about  the  award  recipient,  such 
as  gender  and  socio-economic  status,  and  information  about  the  type  of 
firms  that  received  the  awards,  such  as  number  of  employees  and 
geographic  location.  Much  of  the  data  collected  by  participating  agencies 
are  provided  by  the  SBIR  applicants  at  the  time  they  apply  for  an  award. 
Agencies  provide  additional  information,  such  as  the  grant/contract 
number  and  the  dollar  amount  of  the  award,  after  the  award  is  made. 

For  the  most  part,  all  of  the  eight  agencies  we  reviewed  provided  the 
majority  of  the  SBIR  award  data  requested  by  SBA,  including  the  program 
identification  number;  company  name,  address,  and  contact  information; 
award  year  and  amount;  a  unique  tracking  number  that  will  stay  with  the 
award  through  both  phase  I  and  phase  II;  and  the  title  and  abstract  of  the 
project  funded.  However,  we  also  determined  that  some  of  the  agencies 
are  not  providing  the  full  range  of  information  required  by  SBA.  For 
example,  two  of  the  eight  agencies  we  reviewed  had  not  provided  SBA 
data  on  the  gender  or  socio-economic  status  of  SBIR  award  recipients  in 
2004  and  2005.  Similarly,  in  2005,  five  of  the  eight  agencies  failed  to 
provide  data  on  the  number  of  employees  working  at  the  firms  that 
received  the  awards.  As  a  result,  SBA  does  not  have  information  on  the 
number  of  employees  at  SBIR  awardee  firms  for  about  one-third  of  all  the 
awards  reported  by  these  agencies  in  2004  and  2005. 

SBA  officials  acknowledged  that  agencies  do  not  routinely  provide  all  of 
the  information  requested  because  either  they  do  not  capture  the 
information  in  their  agency  databases  or  they  are  not  requesting  the 
information  from  SBIR  applicants.  Officials  at  the  participating  agencies 
also  cited  other  reasons  for  the  incomplete  data  they  provided  to  SBA.  For 
example,  NIH  officials  stated  that  for  the  past  several  years,  the  SBA  Tech- 
Net  annual  reporting  requirements  have  changed  each  year.  At  the  end  of 
calendar  year  2003,  SBA  changed  the  field  description  from  “minority”  to 
“socially  and  economically  disadvantaged  small  business.”  According  to 
NIH  officials,  because  the  SBIR  information  is  captured  by  the  agency  at 
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the  time  the  application  is  submitted/received,  there  is  a  lag  in  time 
between  when  application  data  is  input  into  the  NIH  database  and  when 
the  agency  receives  new  SBA  data  field  requirements.  According  to  these 
officials,  responding  to  SBA’s  changes  in  field  names  presents  significant 
challenges  to  NIH  for  collecting  the  data  needed  to  complete  the  Tech-Net 
reports,  especially  if  the  requirements  change  annually.  As  a  result,  these 
officials  stated  that  complying  with  the  changing  requirements 
necessitates  significant  NIH  resources  and  efforts  to  research  and  identify 
the  information  needed  for  the  new  data  fields,  not  all  of  which  can  be 
provided.  In  commenting  on  a  draft  of  this  report,  SBA  stated  that  it  has 
only  requested  minor  clarifications  of  data  requirements  and  has  not  made 
frequent  changes  as  stated  by  the  agencies.  SBA  believes  that  this  may 
have  been  caused  by  lack  of  clear  communication  to  the  agencies. 

The  agencies  also  noted  that  data  for  certain  Tech-Net  fields  will  be  absent 
from  their  reports  to  SBA  if  the  data  fields  do  not  exist  in  the  NIH 
application  or  in  its  awards  database.  Similarly,  USDA  officials  stated  that 
although  they  try  to  keep  their  records  as  up-to-date  as  possible,  problems 
occur  when  company  or  contact  information  changes  and  the  SBIR 
recipient  fails  to  provide  updated  information  to  the  agency,  such  as  the  e- 
mail  address  for  the  central  contact  person.  Additionally,  like  NIH,  USDA 
officials  stated  that  certain  information  requested  by  SBA  is  not  collected 
by  their  agency.  For  example,  as  long  as  SBIR  applicants  have  certified 
that  they  meet  the  criterion  of  being  a  small  business  (under  500 
employees),  they  do  not  ask  for  nor  do  they  record  information  on  the 
number  of  employees  in  the  firm. 


Agencies  Generally 
Comply  with  SBA’s 
Formatting  Guidance, 
but  Key  Data  on  SBIR 
Award  Recipients  May 
Be  Inconsistent  or 
Inaccurate 


Participating  agencies  are  providing  some  data  that  do  not  comply  with 
SBA’s  formatting  guidance,  and  while  some  of  these  inconsistencies  are 
corrected  by  SBA’s  quality  assurance  processes,  others  are  not.  As  a  result, 
some  data  elements  in  the  Tech-Net  database  may  be  inconsistent  or 
inaccurate  thereby,  compromising  the  value  of  these  data  for  program 
evaluation.  SBA’s  quality  assurance  efforts  focus  on  obtaining  complete 
and  accurate  data  for  those  fields  essential  to  tracking  specific  awards, 
such  as  the  tracking  number  and  award  amount,  rather  than  on  those 
fields  that  contain  demographic  information  about  the  award  recipient. 
Because  the  data  contained  in  the  public-use  section  of  Tech-Net  will  be 
incorporated  into  the  government-use  section  of  the  database,  inaccurate 
data  in  one  section  of  the  database  will  be  replicated  in  the  other,  and 
these  inaccurate  data  will  limit  evaluations  of  the  SBIR  program. 


Page  10 


GAO-07-38  Small  Business  Innovation  Research 


Both  SBA  and  agency  officials  acknowledge  that  SBA  has  worked  with 
participating  agencies  since  2000  to  help  ensure  greater  consistency  in  the 
formatting  of  the  SBIR  award  data  reported  by  agencies  each  year.  To 
assist  agencies  in  formatting  the  data  and  to  minimize  the  number  of 
inconsistencies  in  the  data  reported  to  SBA,  SBA  has  taken  a  number  of 
steps  to  improve  the  data  formatting  process.  Specifically,  SBA  provided 
all  of  the  participating  agencies  specific  guidance  on  its  requirements  for 
the  data,  including  its  preferred  program  application  for  submitting  the 
data,  the  length  of  each  data  field,  and  whether  data  should  be  entered  as 
numbers,  characters,  or  a  mix  of  both.  SBA  has  also  included  discussions 
of  the  Tech-Net  database  as  an  agenda  item  at  its  quarterly  meetings  with 
SBIR  program  managers  from  each  of  the  participating  agencies. 
Additionally,  SBA  developed  a  reporting  template  for  agencies  to  use  that 
includes  the  required  data  fields  and  instructions  for  appropriate  data 
entry. 

Under  the  current  process,  participating  agencies  aggregate  all  of  the  data 
provided  by  SBIR  recipients  in  their  award  applications  with  additional 
information  on  the  award  amounts  and  submit  the  combined  data  to  SBA 
by  March  15  of  each  year.  SBA  then  electronically  checks  the  data  to 
locate  and  reformat  inconsistencies,  and  adds  the  data  to  the  Tech-Net 
database.  SBA  officials  told  us  that  when  they  detect  inconsistencies  in 
data  fields  essential  to  tracking  a  specific  award,  such  as  the  award 
tracking  number,  contact  information  for  the  recipient  or  principal 
investigator,  or  awarding  agency,  they  contact  the  agency  to  obtain  the 
correct  information.  However,  SBA  does  not  currently  take  steps  to  ensure 
that  agency-provided  data  are  accurate  and  complete.  For  example,  SBA 
does  not  require  agencies  to  submit  a  random  sample  of  applications  so 
that  it  can  compare  the  data  submitted  by  agencies  with  the  original 
applicant  information  to  ensure  that  the  submitted  data  contain  all  the 
relevant  application  information.  Instead,  SBA  relies  on  the  agencies  to 
fully  report  all  the  required  application  information  on  the  awards  they 
make.  SBA  officials  told  us  that  they  believe  that  over  the  past  two  years 
the  quality  and  consistency  of  the  data  received  from  participating 
agencies  had  greatly  improved. 

In  reviewing  the  SBIR  Tech-Net  data  that  the  eight  agencies  reported  to 
SBA  in  2004  and  2005,  we  determined  that  almost  a  quarter  of  the  data 
provided  by  five  agencies  was  incorrectly  formatted  for  one  or  more  fields. 
For  example,  phone  number  and  award  amount  fields  contained  both 
characters  and  numbers  and  first  and  last  names  of  principal  investigators 
were  combined  into  a  single  word  that  was  used  as  both  the  first  and  last 
names.  Moreover,  we  found  that  agency-provided  data  on  gender  and 
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socio-economic  status  for  over  half  of  the  awards  reported  in  2004  were 
incorrectly  formatted.  SBA  officials  said  that  detecting  and  correcting 
some  of  these  formatting  errors  required  a  considerable  amount  of  time 
and  effort,  and  that  their  electronic  check  does  not  detect  all  of  the  errors 
that  we  identified. 

The  formatting  inconsistencies  that  we  identified  arise  primarily  because 
the  SBA  reporting  template  used  by  agencies  to  submit  required  data  can 
be  edited.  Consequently,  agencies  can  and  do  edit  the  template.  Agencies 
can  change  the  names  of  various  data  fields  in  SBA’s  template,  delete 
fields  altogether,  and  enter  data  as  numbers,  characters,  or  both, 
regardless  of  what  SBA  has  specified.  One  of  the  data  format  issues 
identified  by  SBA  was  the  deletion  of  fields  that  agencies  do  not  consider 
relevant  or  necessary.  For  example,  DOD  is  the  only  agency  that  uses  the 
field  called  “branches”  to  specify  which  component  of  DOD,  such  as  the 
Army,  made  the  award;  other  agencies  have  deleted  this  field  because  it 
does  not  relate  to  the  structure  of  their  agency.  Similarly,  we  found 
instances  where  agencies  had  entered  data  on  the  gender  or  socio¬ 
economic  status  of  the  recipient  and  award  amount  in  a  format  that 
differed  from  the  numerical  format  specified  by  SBA.  For  example,  one 
agency  entered  “Y”  and  “N”  rather  than  “0”  and  “1.”  According  to  SBA 
officials,  by  fiscal  year  2007,  this  issue  should  be  resolved  because 
agencies  will  have  to  submit  their  data  through  an  Internet  interface  that 
will  contain  edit  checks  and  should  eliminate  many  of  these  problems. 

We  also  determined  that  inconsistencies  or  inaccuracies  can  arise  in 
certain  data  fields  because  SBA  interprets  the  absence  of  certain  data 
elements  as  a  negative  entry  without  confirming  the  accuracy  of  such  an 
interpretation  with  the  agency.  In  other  words,  if  an  agency  did  not 
provide  information  on  whether  the  recipient  is  a  woman  or  a  member  of  a 
socio-economically  disadvantaged  group,  SBA  has  entered  a  “no”  into  the 
database.  SBA  stated  that  they  generally  do  not  contact  agencies  to  obtain 
correct  information  on  data  elements  that  are  not  used  to  track  specific 
awards,  such  as  gender  or  socio-economic  status  of  the  recipient. 

However,  this  inaccurate  data  on  the  award  recipients  could  limit  efforts 
to  use  these  fields  for  comprehensive  program  evaluation. 

Information  in  the  Tech-Net  database  will  be  used  to  populate  the 
government-use  section  of  the  database  that  SBA  is  developing  (as 
discussed  below)  for  the  purpose  of  supporting  SBIR  program  evaluations. 
However,  SBA  has  no  plans  to  correct  any  of  the  errors  or  inconsistencies 
in  the  database  that  relate  to  the  historical  data  already  collected.  As  a 
result,  the  errors  in  the  existing  database  will  migrate  to  the  government- 
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use  section  of  the  database  and  we  believe  will  compromise  the  usefulness 
of  the  government-use  database  for  program  evaluation  purposes. 


SBA  Is  Five  Years 
Behind  Schedule  in 
Meeting  its  Obligation 
to  Implement  a 
Govemment-U  se 
SBIR  Database 


SBA  has  not  met  its  obligation  to  implement  a  restricted  government-use 
database  that  would  allow  SBIR  program  evaluation  as  directed  by  the 
2000  SBIR  reauthorization  act.  As  outlined  in  the  legislation,  SBA,  in 
consultation  with  federal  agencies  participating  in  the  SBIR  program,  was 
to  develop  a  secure  database  by  June  2001  and  maintain  it  for  program 
evaluation  purposes  by  the  federal  government.  SBA  planned  to  meet  this 
requirement  by  expanding  the  existing  Tech-Net  database  to  include  a 
restricted  government-use  section  that  would  be  accessible  only  to 
government  agencies  and  other  authorized  users.  In  constructing  the 
government-use  section  of  the  database,  SBA  planned  to  supplement 
existing  data  already  gathered  for  the  public-use  section  of  the  Tech-Net 
database  with  information  from  SBIR  recipients  and  from  participating 
agencies  on  commercialization  outcomes  for  phase  II  SBIR  awards. 
However,  according  to  SBA  officials,  the  agency  has  been  unable  to  meet 
this  requirement,  primarily  because  of  increased  security  and  other 
information  technology  project  requirements,  agency  management 
changes,  and  budgetary  constraints.  SBA’s  current  goal  for  having  the 
government-use  section  of  the  Tech-Net  database  operational  is  October  1, 
2006.  In  commenting  on  a  draft  of  this  report,  SBA  modified  this  date, 
stating  that  they  anticipate  having  the  government-use  section  of  the  Tech- 
Net  database  operational  early  in  fiscal  year  2007. 


To  date,  with  the  help  of  two  contractors,  SBA  has  developed  the 
framework  for  importing  data  into  the  government-use  section  of  the 
Tech-Net  database  and  for  an  Internet-based  interface  that  would  allow 
agencies  and  award  recipients  to  access  the  database  and  enter 
commercialization  information.  According  to  SBA  officials,  as  currently 
envisioned,  the  government-use  section  of  the  Tech-Net  database  will 
include  the  records  of  all  applicants,  including  those  that  did  not  receive 
SBIR  awards.  Participating  SBIR  agencies  will  be  asked  to  provide  a 
unique  business  identification  number,  called  the  Data  Universal 
Numbering  System  or  DUNS  number,  for  each  award  recipient, 
information  about  SBIR  applicants  that  were  not  funded,  and  any 
historical  data  they  have  obtained  about  the  commercialization  of  SBIR 
funded  technologies.  SBA  has  developed  a  standardized  electronic 
commercialization  questionnaire  to  gather  data  for  the  government-use 
section  of  the  database  from  applicants  and  award  recipients.  Information 
that  will  be  captured  in  the  questionnaire  will  include  the  number  of  SBIR 
awards  the  company  has  received,  the  number  of  patents  or  copyrights 
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that  have  resulted  from  the  award,  sales  revenue  realized  as  a  result  of  the 
SBIR  award,  and  sources  of  additional  investment  funding.  SBA  officials 
told  us  that  the  commercialization  questionnaire  will  become  an  integral 
part  of  the  SBIR  application  process  in  the  future,  and  any  company 
applying  for  an  SBIR  award  will  be  required  to  complete  and  update  the 
relevant  information  on  phase  II  SBIR  awards  previously  received  by  the 
company  at  the  time  the  application  is  submitted.  According  to  SBA 
officials,  applications  will  not  be  accepted  until  this  information  is 
completed,  and  failure  to  submit  the  information  may  affect  an  applicant’s 
ability  to  receive  an  award.  In  addition,  SBIR  award  recipients  will  be 
requested  to  voluntarily  update  the  commercialization  information  in  the 
government-use  section  of  the  Tech-Net  database  armually  for  a  minimum 
period  of  five  years  following  the  completion  of  their  SBIR-funded  project. 

Although  SBA  has  developed  the  majority  of  the  functions  needed  to 
populate  the  government-use  section  of  the  Tech-Net  database  with  the 
data  currently  gathered  for  the  public-use  section  of  the  database,  it  can 
not  be  made  operational  until  certain  security  issues  are  resolved.  For 
example,  because  the  government-use  section  of  the  database  will  contain 
information  from  recipients  that  is  considered  proprietary  and 
confidential,  SBA  needs  to  ensure  that  adequate  security  measures  are  in 
place  to  prevent  unauthorized  access  to  the  data.  This  entails  the 
successful  completion  of  a  series  of  security  and  development  checks  to 
ensure  that  the  database  system  is  operating  as  designed.  While  SBA 
officials  expect  the  government-use  section  of  the  database  to  be 
operational  by  October  1,  2006,  they  also  recognize  that  additional 
enhancements,  such  as  improving  the  user-friendliness  of  the  interface  for 
online  submission  of  SBIR  data  by  participating  agencies  and  recipients, 
will  be  needed  after  the  system  is  made  operational.  According  to  SBA 
officials,  the  agency’s  priority  is  to  get  the  government-use  section  of  the 
database  up  and  running  before  considering  further  improvements  to  the 
database. 
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Tracking  Data  on  the 
Commercialization 
Success  of  SBIR 
Projects  Varies 
Among  Agencies  and 
Remains  a  Challenge 


Seven  of  the  eight  participating  agencies  we  reviewed  have  implemented 
techniques  to  track  the  commercialization  success  of  their  SBIR-funded 
projects,  and  the  eighth  is  planning  to  do  so,  although  the  methodological 
rigor  of  these  techniques  varies  significantly.  Under  the  program’s 
authorizing  legislation  and  SBA’s  implementation  guidance,  agencies  have 
been  given  considerable  flexibility  to  design,  monitor,  and  evaluate  the 
extent  to  which  their  SBIR  programs  have  achieved  commercialization 
success.  For  example,  while  some  agencies  use  more  systematic 
approaches  to  gathering  data,  such  as  periodically  surveying  SBIR  award 
recipients,  other  agencies  are  less  methodical,  choosing  instead  to  follow 
up  periodically  with  a  relatively  small  sample  of  SBIR  award  recipients. 
Regardless  of  how  they  track  commercialization  success,  both  SBA  and 
agency  officials  generally  agree  that  several  factors  complicate  their 
efforts  to  obtain  this  information. 


Agencies’  Efforts  to  Track 
SBIR  Commercialization 
Success  Vary 


Of  the  eight  agencies  we  reviewed,  five  systematically  and  periodically 
survey  SBIR  recipients  to  gather  a  variety  of  data  on  program 
participation,  including  the  recipients’  commercialization  experiences. 
Specifically, 


•  Since  2000,  DOD  has  systematically  gathered  information  electronically  on 
the  commercialization  of  phase  II  awards  from  all  phase  I  and  phase  II 
applicants  and  award  recipients  and  maintains  the  information  in  a 
commercialization  database.  Commercialization  outcomes  that  DOD 
monitors  include  such  measures  as  (1)  sales  revenue  from  new  products 
and  non-R&D  services  resulting  from  the  phase  II  technology;  (2) 
additional  investment  from  sources  other  than  the  federal  SBIR  program 
in  activities  that  further  the  development,  commercialization,  or  both  of 
the  phase  II  technology;  (3)  whether  the  phase  II  technology  has  been  used 
in  a  DOD  system  or  acquisition  program,  and  if  so,  which  system  or 
program;  (4)  the  number  of  patents  resulting  from  the  contractor’s 
participation  in  the  SBIR  program;  (5)  growth  in  the  number  of  employees 
at  the  firm;  and  (6)  whether  the  firm  has  completed  an  initial  public 
offering  of  stock  resulting,  in  part,  from  the  phase  II  project.  DOD  uses  the 
accumulated  data  to  assign  a  commercialization  score  to  applicants  that 
have  received  four  or  more  prior  SBIR  awards  based  on  a  comparison  of 
their  commercialization  experience  with  the  average  experience  of  other 
comparable  applicants  and  uses  the  score  to  help  select  proposed  projects 
for  funding.  In  addition,  recipients  of  phase  II  awards  are  required  to 
update  the  information  one  year  after  the  start  of  the  project,  at  the 
completion  of  the  project,  and  subsequently  when  the  recipient  submits  a 
new  SBIR  application  to  DOD.  Firms  that  do  not  submit  a  new  SBIR 
application  are  asked  to  voluntarily  provide  updates  on  an  annual  basis 
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after  the  completion  of  their  phase  II  project.  According  to  DOD  officials, 
66  percent  of  phase  II  award  recipients  updated  their  commercialization 
information  when  they  submitted  a  new  application,  11  percent  provided 
the  information  without  submitting  a  new  application,  and  23  percent  did 
not  update  their  information. 

•  For  over  23  years,  DOE  has  conducted  an  annual  survey  of  SBIR  phase  II 
recipients,  both  active  and  inactive,  to  track  the  commercialization 
success  of  its  SBIR-funded  projects.  The  survey  requests  recipients  to  (1) 
list  all  products  and  services  derived  from  their  SBIR  projects;  (2)  report 
on  sales,  phase  III  investment  related  to  these  products  and  service,  or 
both;  and  (3)  identify  which  phase  II  projects  contributed  to  the 
development  of  the  products  and  services.  According  to  DOE, 
approximately  90  percent  of  its  phase  II  recipients  respond  to  these  annual 
surveys. 

•  NASA  has  systematically  gathered  information  on  the  commercialization 
of  SBIR  awards  through  annual  surveys  of  phase  II  award  recipients  from 
1997  to  2002.  In  these  surveys,  NASA  obtained  data  on  various  commercial 
outcomes,  such  as  sales  to  nongovernment  markets  of  the  SBIR-funded 
research  results,  procurement  of  the  research  results  by  NASA  or  other 
federal  agencies,  cumulative  private  capital  funding,  royalty  and  licensing 
revenue  from  nongovernment  sources  based  on  the  SBIR-funded  research 
results,  creation  of  new  business  ventures  based  on  the  SBIR-funded 
research  results,  and  number  of  patents  and  patent  applications  resulting 
from  these  awards.  According  to  NASA,  approximately  91  percent  of  its 
phase  II  award  recipients  responded  to  these  annual  surveys. 

•  NIH  also  surveys  SBIR  award  recipients  to  gather  commercialization 
information.  Specifically,  in  2002,  NIH  conducted  a  comprehensive 
evaluation  of  its  SBIR  program.  As  part  of  this  evaluation,  NIH  surveyed 
recipients  of  phase  II  awards  between  1992  and  2001  to  obtain  data 
relating  to  the  range  of  SBIR  program  goals,  including  commercialization 
of  research  results.  According  to  NIH  officials,  the  2002  survey  results 
formed  a  baseline  that  NIH  staff  could  use  to  systematically  monitor  and 
evaluate  the  program.  In  2004  and  2005,  NIH  again  contacted  award 
recipients  to  update  the  information  obtained  in  the  2002  survey. 
Commercialization  outcomes  tracked  by  NIH  include  (1)  sales  realized  for 
a  product  or  service  that  resulted  from  the  SBIR-funded  research;  (2) 
status  of  the  Food  and  Drug  Administration  (FDA)  approval  process  for 
the  SBIR-funded  research  results;  (3)  receipt  of  FDA  approval  for  SBIR- 
funded  research  results;  and  (4)  receipt  of  additional  non-SBIR  funding  or 
capital. 
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•  Since  about  1998,  NSF  has  collected  historical  commercialization 
information  from  all  phase  II  award  recipients  at  the  time  an  SBIR 
application  is  submitted.  This  information  is  used  in  the  proposal  review 
process  to  help  select  proposed  projects  for  funding.  In  2005,  NSF 
developed  an  annual  survey  of  phase  II  award  recipients  that  will  be  used 
to  gather  information  three,  five,  and  eight  years  following  their  awards. 
Specific  outcomes  on  which  NSF  will  gather  data  include  sales  revenue 
based  on  the  SBIR-funded  research  results,  growth  of  overall  company 
sales  and  employment,  receipt  of  additional  non-SBIR  funding,  and  patents 
related  to  the  SBIR  funded  research. 

In  contrast,  two  of  the  remaining  three  agencies  we  reviewed  have  focused 
their  efforts  on  gathering  anecdotal  success  stories  and  conducting 
periodic  follow-up  with  a  relatively  smaller  sample  of  SBIR  award 
recipients.  For  example,  over  the  past  7  years,  ERA  has  contacted  all 
phase  II  award  recipients  after  their  projects  end  to  learn  about  their 
commercial  successes.  Based  on  these  contacts,  officials  estimate  that 
approximately  25  percent  of  the  phase  II  projects  funded  by  EPA  have 
been  commercialized.  EPA  defines  “success”  as  the  receipt  of  more  than 
$300,000  in  revenue  from  sales  of  the  SBIR-funded  project,  an  amount 
greater  than  the  SBIR  funds  awarded  by  EPA.  Similarly,  USDA  has 
periodically  contacted  a  sample  of  about  20  to  25  percent  of  award 
recipients  to  obtain  information  about  sales  of  their  SBIR-funded  research 
results.  USDA  last  surveyed  its  phase  II  award  recipients  in  1997.  USDA 
publishes  the  success  stories  on  its  Web  site  and  in  an  agency  newsletter. 
About  2,500  people  receive  the  newsletter  and  USDA  makes  copies  of  the 
success  stories  available  at  SBIR  conferences.  According  to  USDA 
officials,  in  the  future  they  hope  to  gather  data  more  systematically  and 
conduct  site  visits  to  the  SBIR  firms.  The  eighth  agency  we  reviewed, 

NIST,  has  recently  developed  a  Web-based  survey  that  it  plans  to  send  in 
2006  to  all  of  its  SBIR-award  recipients. 


Tracking  the 
Commercialization 
Success  of  the  Overall 
SBIR  Program  Remains  a 
Challenge 


Although  each  of  the  eight  agencies  we  reviewed  has  implemented  or 
plans  to  implement  a  method  for  gathering  commercialization  data,  agency 
officials  identified  several  factors  that  complicate  their  efforts.  First, 
agency  officials  stated  that  it  is  difficult  to  track  commercialization 
because  it  can  take  years  before  companies  achieve  commercial  success. 
For  example,  USDA  officials  stated  that,  even  over  the  short  term,  the 
effort  to  contact  past  award  recipients  consumes  considerable  effort. 
During  this  time,  companies  may  move,  change  names,  start  a  new 
business,  or  be  purchased  by  other  firms,  all  of  which  make  it  difficult  for 
the  agencies  to  track  and  link  companies  to  the  original  SBIR  awards. 
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Second,  because  the  authorizing  legislation  lacks  a  clear  definition  of  what 
constitutes  “commercialization”  success,  agencies  not  only  differed  on  the 
types  of  commercialization  outcomes  they  captured,  but  also  in  how  they 
tracked  commercial  success.  SBA  officials  acknowledged  that  its  guidance 
has  provided  considerable  latitude  to  agencies  on  this  issue  in  light  of  the 
wide  range  of  industries  represented  by  the  participating  agencies. 
Commercialization  outcomes  captured  by  the  participating  agencies 
included  sales  revenue  based  on  the  SBIR-funded  research  results,  receipt 
of  additional  non-SBIR  funding  to  further  develop  the  research  results, 
marketing  activities  ongoing  or  completed,  and  public  offering  of  company 
stock.  However,  not  all  agencies  are  tracking  all  of  these  outcomes; 
therefore,  assessing  overall  commercial  success  of  the  SBIR  program 
across  the  various  agencies  remains  a  challenge. 

Third,  agency  officials  stated  that  SBIR  award  recipients  may  be  reluctant 
to  provide  information  related  to  their  trade  and  business  operations, 
which  they  consider  proprietary  and  sensitive.  Companies  are  often  not 
willing  to  provide  comprehensive  data  on  their  sales  and  particularly  the 
investments  they  receive  due  to  competitive  concerns.  Finally,  agency 
officials  told  us  that  past  recipients  have  no  incentive  to  voluntarily 
complete  commercialization  surveys  and  update  the  information  on  their 
commercial  experience  unless  they  are  applying  for  a  new  SBIR  award.  As 
a  result,  they  do  not  expect  that  a  large  percentage  of  recipients  will 
complete  the  information  needed  for  the  government-use  section  of  the 
Tech-Net  database.  Agency  officials  believe  that  despite  their  best  efforts, 
the  data  needed  to  conduct  evaluations  of  the  SBIR  program  are  likely  to 
be  incomplete. 


Conclusions 


In  the  last  5  years,  SBA  has  been  unable  to  meet  the  congressional 
directive  to  develop  a  government-use  database  that  would  provide  better 
information  on  the  SBIR  program  and  allow  for  program  evaluation. 
Although  it  has  established  a  public-use  Tech-Net  database  and  has 
worked  with  participating  agencies  to  achieve  greater  consistency  in  the 
data  submitted  for  the  database,  the  quality  of  the  data  remains  a  concern. 
The  steps  on  which  SBA  relies  to  ensure  that  data  are  complete  and 
accurate  are  inadequate  and  it  has  no  plans  to  correct  errors  or  supply 
missing  data  associated  with  the  historical  data  already  in  the  database. 

We  believe  that  imless  necessary  controls  are  established  and 
implemented  to  ensure  the  completeness,  consistency,  and  accuracy  of  the 
SBIR  data  reported  to  SBA  by  participating  agencies,  the  government-use 
section  of  the  Tech-Net  database,  which  depends  on  the  public-use  Tech- 
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Net  database,  will  be  of  limited  use  for  program  evaluation  purposes  when 
it  becomes  operational. 

Recommendation  for 
Executive  Action 

We  recommend  that  the  Administrator,  SBA,  and  the  SBIR  participating 
agencies  work  together  to  strengthen  efforts  to  ensure  that  the  data 
collected  for  SBA’s  Tech-Net  database  are  complete,  consistent,  and 
accurate. 

Agency  Comments 
and  Our  Evaluation 

We  provided  SBA  and  the  eight  SBIR  participating  agencies  included  in 
this  review  a  draft  of  this  report  for  their  review  and  comment.  SBA  and 
the  eight  agencies  generally  agreed  with  the  report’s  findings  and  SBA  and 
five  of  the  eight  agencies  also  stated  their  concurrence  with  the 
recommendation.  Three  agencies — ERA,  NIST,  and  NSF — did  not  indicate 
whether  they  agreed  or  disagreed  with  the  recommendation. 

In  addition,  SBA  stated  that  it  was  concerned  that  our  conclusions  did  not 
reflect  the  fact  that  it  plans  to  have  the  government-use  section  of  the 
Tech-Net  database  operational  by  early  fiscal  year  2007.  We  have  not 
modified  our  conclusions  because  the  fact  remains  that  SBA  has  not  met 
the  congressional  directive  to  establish  a  government-use  database  during 
the  last  five  years.  Moreover,  throughout  this  review,  SBA  officials  told  us 
that  the  database  would  be  operational  by  October  1,  2006.  However,  in  its 
official  comments,  the  agency  has  again  modified  this  date  to  some  time 
early  in  fiscal  year  2007.  SBA  also  provided  us  with  technical  comments 
that  we  have  incorporated  as  appropriate.  SBA’s  letter  is  included  in 
appendix  II  and  the  letters  that  we  received  from  DOD,  NASA,  and  NIH  are 
included  in  appendix  III  through  V. 

As  agreed  with  your  office,  unless  you  publicly  announce  the  contents  of 
this  report  earlier,  we  plan  no  further  distribution  until  30  days  from  the 
report  date.  At  that  time,  we  will  send  copies  of  this  report  to  interested 
congressional  committees;  the  Administrators  of  the  Environmental 
Protection  Agency,  National  Aeronautics  and  Space  Administration,  and 
Small  Business  Administration;  the  Directors  of  the  National  Institutes  of 
Health,  National  Institute  of  Standards  and  Technology,  and  National 
Science  Foundation;  the  Secretaries  of  Agriculture,  Defense,  and  Energy; 
and  other  interested  parties.  We  will  also  make  copies  available  to  others 
upon  request.  In  addition,  the  report  will  be  available  at  no  charge  on  the 
GAO  Web  site  at  http://www.gao.gov. 
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If  you  or  your  staff  have  any  questions  about  this  report,  please  contact  me 
at  (202)  512-3841  or  mittala@gao.gov.  Contact  points  for  our  Offices  of 
Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page 
of  this  report.  GAO  staff  who  made  contributions  to  this  report  are  listed 
in  appendix  VI. 


Anu  K.  Mittal 

Director,  Natural  Resources 
and  Environment 
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Appendix  I:  Objectives,  Scope  and  Methods 


Our  objectives  for  this  review  were  to  identify  (1)  the  types  of  data  that 
participating  Small  Business  Innovation  Research  (SBIR)  program 
agencies  are  reporting  to  the  Small  Business  Administration  (SBA)  for 
inclusion  in  the  Tech-Net  database,  (2)  the  extent  to  which  agencies 
provide  data  for  the  Tech-Net  database  in  a  standard  format  to  enable 
program  evaluation,  (3)  the  extent  to  which  SBA  has  met  the  mandate  to 
establish  by  early  2001  a  government-use  database  that  can  be  used  for 
program  evaluation,  and  (4)  the  extent  to  which  participating  SBIR 
agencies  have  developed  and  implemented  techniques  to  track  the 
commercialization  success  of  SBIR  projects. 

In  conducting  this  study,  we  reviewed  SBA  and  the  SBIR-related  activities 
of  8  of  the  11  SBIR  participating  agencies — Department  of  Defense  (DOD), 
Department  of  Energy  (DOE),  Environmental  Protection  Agency  (EPA), 
National  Aeronautics  and  Space  Administration  (NASA),  National 
Institutes  of  Health  (NIH),  National  Institute  of  Standards  and  Technology 
(NIST),  National  Science  Foundation  (NSF),  and  Department  of 
Agriculture  (USD A).  These  eight  agencies  account  for  over  98  percent  of 
the  total  dollars  awarded  by  the  program  in  fiscal  year  2005.  To  determine 
the  types  of  data  these  participating  agencies  are  reporting  to  SBA  and  the 
extent  to  which  SBA  has  complied  with  the  requirement  to  establish  a 
government-use  database  that  can  be  used  for  program  evaluation 
purposes,  we  compared  the  provisions  in  the  Small  Business  Innovation 
Research  Program  Reauthorization  Act  of  2000,  relevant  legislative 
histories,  SBA’s  Policy  Directive  implementing  the  act  issued  in  September 
2002,  and  other  guidance  with  agency  SBIR  Tech-Net  database  reports.  We 
identified  and  interviewed  SBIR  program  officials  at  each  agency  and 
officials  responsible  for  submitting  program  data  to  SBA.  For  these 
interviews,  we  used  a  protocol  guide  to  obtain  information  on  program 
operations,  data  reporting,  data  quality,  and  the  Tech-Net  database, 
including  development  of  the  government-use  section  of  the  database.  We 
also  reviewed  documentation  from  each  of  the  eight  agencies  and  SBA 
regarding  data  elements  and  formatting  instructions. 

To  determine  the  extent  to  which  data  for  the  Tech-Net  database  are 
provided  in  a  standard  format,  enabling  program  evaluation,  we  compared 
data  provided  to  SBA  by  the  eight  participating  agencies  with  data  in 
SBA’s  Tech-Net  database  for  fiscal  years  2004  and  2005,  the  2  most  recent 
years  for  which  data  were  available.  The  agency  data  included  information 
about  the  award  itself,  such  as  the  award  amount,  a  descriptive  abstract  of 
the  project,  and  a  unique  tracking  number;  information  about  the  award 
recipient,  such  as  gender  and  socio-economic  status;  and  information 
about  the  type  of  firm  that  received  the  award,  such  as  number  of 
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employees  and  geographic  location.  To  assess  the  reliability  of  the  data 
provided  by  the  agencies,  we  reviewed  SBA’s  data  system,  internal 
controls,  and  related  guidance,  rather  than  the  systems  and  internal 
controls  participating  agencies  use  to  create  the  data  provided  to  SBA.  To 
assess  the  reliability  of  the  data  in  SBA’s  Tech-Net  database,  we 
interviewed  SBA  officials  about  the  database  and  reviewed  related 
documentation.  We  also  conducted  tests  of  the  data  themselves.  We  used 
analytic  software  to  compare  the  data  provided  by  participating  agencies 
with  the  data  maintained  by  SBA.  We  focused  our  review  on  data  fields 
considered  critical  by  SBA  officials,  such  as  the  awarding  agency,  the  date 
of  award,  the  award  recipient,  the  amount  of  the  award,  and  the  purpose 
of  the  award.  We  also  reviewed  data  fields  related  to  SBIR  program  goals, 
such  as  gender  and  socio-economic  status  of  the  recipient,  and  data  on  the 
number  of  employees  at  the  recipient  firm.  We  used  GAO’s  data  reliability 
guidance  to  identify  key  attributes  of  data  quality  that  facilitate  program 
evaluation.  These  attributes  include  completeness,  accuracy,  and 
consistency  in  format.  Finally,  we  reviewed  internal  quality  control 
procedures.  We  determined  that  the  data  are  sufficiently  reliable  for  our 
purpose,  which  is  to  report  on  SBA’s  efforts  to  ensure  consistency  and 
completeness  of  the  data  it  receives. 

To  determine  the  extent  to  which  the  government-use  database 
requirements  have  been  implemented,  we  interviewed  SBA  officials,  and 
reviewed  related  documentation,  such  as  minutes  from  meetings  of  SBA 
and  SBIR  program  directors,  and  overviews  of  the  existing  and  planned 
data  systems.  In  addition,  we  attended  a  demonstration  of  the  proposed 
Internet  interface  for  the  government-use  section  and  interviewed  the 
current  contractor  assisting  SBA  about  implementation  progress.  We  also 
reviewed  a  contractor-prepared  analysis  of  the  functional  and  data 
requirements  for  the  integration  of  the  public-  and  government-use 
sections  of  the  Tech-Net  database.  At  each  of  the  eight  participating 
agencies,  we  interviewed  SBIR  officials  regarding  the  extent  to  which  SBA 
had  consulted  them  in  the  development  of  the  government-use  database. 

To  determine  the  extent  to  which  participating  agencies  have  developed 
and  implemented  techniques  to  evaluate  commercialization  success  of 
SBIR  projects,  we  reviewed  agencies’  documentation  about  their 
commercialization  assistance  and  monitoring  efforts.  Specifically,  we 
reviewed  surveys  that  agencies  had  administered  to  award  recipients, 
resulting  reports  on  survey  results,  and  anecdotal  descriptions  of 
commercialization  success.  We  also  reviewed  provisions  in  SBIR 
legislation,  relevant  legislative  histories,  and  SBA’s  Policy  Directive 
regarding  commercialization  of  SBIR-funded  projects,  as  well  as  past  GAO 
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reports.  In  addition,  we  interviewed  officials  at  each  of  the  eight 
participating  agencies  to  obtain  information  on  the  specific 
commercialization  outcomes  they  monitor,  the  history  of  each  agency’s 
data  collection  efforts,  and  the  agencies’  experience  in  obtaining  such 
information  from  current  and  past  award  recipients. 

We  conducted  our  work  from  April  2006  through  September  2006  in 
accordance  with  generally  accepted  government  auditing  standards. 
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Note:  GAO  comments 
supplementing  those  in 
the  report  text  appear  at 
the  end  of  this  appendix. 


See  comment  1 . 
See  comment  2. 

See  comment  3. 
See  comment  4. 


Page  numbers  in  draft 
report  may  differ  from 
those  in  this  report. 


U.S.  Small  Business  Administration 

Washington,  DC  204  I  6 


October  6,  2006 


Ms.  Anu  Mittal 

Director,  Natural  Resources  and  Environment 
U.S.  Government  Accountability  Office 
441  G  St,  NW 
Washington,  DC  20548 

Dear  Ms.  Mittal: 

Thank  you  for  the  opportunity  to  review  and  comment  on  the  Government  Accountablity 
Office  (GAO)  draft  report  entitled,  “Small  Business  Innovation  Research:  Agencies  Need 
to  Strengthen  Efforts  to  Improve  the  Completeness.  Consistency,  and  Accuracy  of 
Awards  Data.”  GAO-07-38.  We  have  reviewed  the  draft  report  and  have  the  following 
comments: 

-Footnote  2  of  the  report  is  confusing.  While  it  does  indicate  there  are  eleven 
SBIR  participating  agencies,  the  language  of  the  footnote  is  uneven. 

-On  page  4,  the  report  mentions  that  the  SBIR  Policy  Directive  (PD)  was  issued  in 
September  2002.  This  should  be  reworded  to  state  that  the  amended  PD  was 
issued  in  September  2002,  otherwise,  someone  reading  it  might  think  2002  was 
the  first  time  SBA  issued  a  PD. 

-Footnote  7  is  confusing  -  read  literally,  it  looks  like  the  SBA  is  an 
SBIR  participating  agency. 

-Pages  5  and  1 1  of  the  report  state  that  the  SBIR  participating  agencies  have  cited 
a  “variety”  of  reasons  for  not  providing  the  data  requested  including  “frequent” 
changes  in  SBA’s  data  requirements.  SBA  contends  that  there  have  not  been 
“frequent”  changes,  but  rather  minor  clarifications  of  data  requirements. 

For  example,  NIH  cited  a  change  in  SBA’s  reporting  requirements  as  a  reason  for 
submission  of  incomplete  data.  SBA  changed  the  field  name  from  “Minority”  to 
“Socially  and  Economically  Disadvantaged”  because  it  is  the  accepted  description 
today.  The  content  of  that  field  was  not  expected  to  change.  Perhaps  our 
communication  about  that  was  not  clear,  but  it  is  not  a  new  field. 

-Page  16  discusses  the  SBIR  participating  agencies’  efforts  in  tracking  the 
commercialization  success  of  the  SBIR  awardee.  The  report  discusses  the 
differences  in  how  the  agencies  define  commercialization  success.  The  SBA 
looks  to  assist  in  resolving  this  problem  by  using  the  Government  Database  to 
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See  comment  5. 


assess  commercialization  program-wide. 

As  a  whole,  the  general  findings  and  recommendations  of  the  report  appear  reasonable. 
We  do,  however,  take  exception  to  the  tone  and  content  of  the  “Conclusions”  section  of 
the  report.  Specifically,  we  are  concerned  that  while  the  “Conclusion”  indicates  that 
SBA  has  not  developed  the  Government-Use  section  of  the  TechNet  Database  in  over  six 
years,  it  does  not  mention  that  SBA  anticipates  having  this  section  of  the  database 
operational  early  in  FY  2007. 

Additionally,  while  we  concur  with  the  finding  that  the  quality  of  the  data  remains  a 
concern,  SBA  has  taken  significant  steps  to  help  to  improve  the  quality  of  the  data.  SBA 
is  working  with  the  agencies  to  improve  the  controls  on  the  propriety  of  the  data 
submitted. 

In  the  section  labeled  “Conclusions,”  the  report  states  “The  steps  that  SBA  relies  on  to 
ensure  that  data  are  complete  and  accurate  are  inadequate  and  it  has  no  plans  to  correct 
errors  or  supply  missing  data  associated  with  historical  data  already  in  the  database.” 
SBA  is  aware  of  records  in  the  database  that  are  incomplete.  However,  this  can  be 
rectified  by  using  a  combination  of  electronic  and  manual  processes  to  gather  the 
missing  data  elements.  The  collection  of  DUNS  numbers  in  the  database  would  allow 
SBA  to  access  information  provided  by  many  awardees  to  the  Central  Contractor 
Registration  (CCR)  database  and  the  Dynamic  Small  Business  (DSBS)  database.  DUNS 
matching  would  allow  the  collection  of  current  firm-level  information.  SBA  and  the 
participating  agencies  will  have  to  work  together  to  correct  some  of  the  historical  data. 
SBA  has  spoken  with  one  agency  to  date  about  working  cooperatively  to  provide 
historical  data  for  older  awards  and  has  a  commitment  from  that  agency.  Similar  dialog 
and  success  can  be  expected  with  other  agencies. 

Should  you  have  questions  on  these  comments,  please  contact  Edsel  Brown,  Assistant 
Administrator,  Office  of  Technology  at  (202)  205-7343. 

Sincerely, 


Acting  Associate 
Office  of  Government  Contracting 


Administrator 
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The  following  are  GAO’s  comments  on  the  Small  Business 
Administration’s  (SBA)  letter  dated  October  6,  2006. 


GAO  Comments 


We  revised  footnote  2  to  more  clearly  reflect  the  11  SBIR  participating 
agencies. 


2.  We  revised  the  text  to  clarify  this  statement. 


3.  We  deleted  reference  to  SBA  in  footnote  7. 


4.  We  added  SBA’s  position  to  the  report. 

5.  We  added  SBA’s  position  to  the  report.  However,  because  SBA  has  not 
met  the  congressional  deadline  to  develop  a  government-use  database, 
we  did  not  modify  our  conclusions. 
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OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON.  DC  20301-3000 


ACQUISITION 
TECHNOLOGY 
ANC3  LOGISTICS 


CCT  96  2m 


Ms.  Anu  K.  Mittal 

Director,  Natural  Resources  and  Environment 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Ms.  Mittal: 

This  is  the  Department  of  Defense  (DoD)  response  to  GAO  Draft  Report, 
“SMALL  BUSINESS  INNOVATION  RESEARCH:  Agencies  Need  to  Strengthen 
Efforts  to  Improve  the  Completeness,  Consistency,  and  Accuracy  of  Awards  Data,” 
(GAO  Code  360677/  GAO-07-38),  dated  September  20,  2006. 

The  GAO  recommends  that  the  Administrator,  the  Small  Business  Administration 
(SBA)  and  agencies  participating  in  the  SBIR  Program  work  together  to  strengthen 
efforts  to  ensure  that  the  data  collected  for  SBA’s  Tech-Net  database  are  complete, 
consistent,  and  accurate.  The  Department  concurs  with  the  recommendation  recognizing 
that  complete,  high-integrity  data  is  essential  for  program  monitoring,  assessment  and 
evaluation. 

We  appreciate  the  opportunity  to  comment  on  the  draft  report  and  look  forward  to 
working  in  concert  with  the  SBA  and  the  other  participating  agencies  to  improve  the 
quality  of  Tech-Net  data. 


Sincerely, 


OiLo<j^  -fe:' 


Frank  M.  Ramos 

Director,  Small  Business  Programs 


Enclosure: 
As  stated 
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GAO  DRAFT  REPORT  -  DATED  SEPTEMBER  20, 2006 
GAO  CODE  360677/GAO-07-38 

“SMALL  BUSINESS  INNOVATION  RESEARCH:  Agencies  Need  to  Strengthen 
Efforts  to  Improve  the  Completeness,  Consistency,  and  Accnracy  of  Awards  Data” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  RECOMMENDATION 

RECOMMENDATION :  The  GAO  recommends  that  the  Administrator,  Small 
Business  Administration  (SBA)  and  agencies  participating  in  the  SBIR  Program  work 
together  to  strengthen  efforts  to  ensure  that  the  data  collected  for  SBA’s  Tech-Net 
database  are  complete,  consistent,  and  accurate. 

Response:  Concur.  Complete,  high-integrity  data  is  essential  for  SBIR  program 
monitoring,  assessment  and  evaluation.  The  Department  fully  supports  SBA  annual  data 
calls  and  looks  forward  to  working  together  with  the  SBA  and  the  other  participating 
agencies  to  improve  the  quality  of  Tech-Net  data. 
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National  Aeronautics  and 
Space  Administration 

Headquarters 

Washington,  DC  20546-0001 


Innovative  Partnerships  Program  Office 

Reply  to  Attn  of: 

Ms.  Ann  K.  Mittal 

Director,  Natural  Resources  and  Environment 
United  States  Government  Accountability  Office 
Washington,  D.C.  20548 


Dear  Ms.  Mittal: 

NASA  appreciates  the  opportunity  to  comment  on  the  draft  Government  Accountability 
Office  (GAO)  report,  “Small  Business  Innovation  Research  Agencies  Need  to  Strengthen 
Efforts  to  Improve  the  Completeness,  Consistency,  and  Accuracy  of  Awards  Data” 
(GAO-07-38). 

The  draft  report  makes  a  single  recommendation,  addressed  to  all  Small  Business 
Innovation  Research  (SBIR)  participating  agencies,  including  NASA  and  the  Small 
Business  Administration  (SBA),  which  is  responsible  for  providing  policy  direction  for 
the  program  across  all  agencies.  The  report  recommends  “that  the  Administrator,  SBA, 
and  the  SBIR  participating  agencies  work  together  to  strengthen  efforts  to  ensure  that  the 
data  collected  for  SBA’s  Tech-Net  database  are  complete,  consistent,  and  accurate.” 

NASA  concurs  with  this  recommendation.  The  SBIR  program  executive  and  program 
manager  responsible  for  the  execution  of  the  program  at  NASA  will  contact 
Mr.  Edsel  M.  Brown  Jr.,  Assistant  Administrator,  Office  of  Technology,  U.S.  Small 
Business  Administration,  to  affirm  NASA’s  commitment  to  participate  in  joint 
discussions  with  all  Federal  agencies  involved  for  the  purposes  of  considering  GAO’s 
findings  and  recommendation,  as  well  as  determining  and  implementing  an  appropriate 
course  of  action. 

Thank  you  for  the  opportunity  to  respond  to  this  draft  report. 


c 


Innovative  Partnerships  Program  Office 
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DEPARTMENT  OF  HEALTH  &  HUMAN  SERVICES 


Public  Health  Service 


National  Institutes  of  Health 
Bethesda,  Maryland  20892 


6  2001? 


Ms.  Anu  K.  Mittal 

Director,  Natural  Resources  and  Environment 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  D.C.  20548 

Dear  Ms,  Mittal: 

We  appreciate  the  opportunity  to  review  and  comment  on  the  U.S.  Government 
Accountability  Office  (GAO)  report.  Small  Business  Innovation  Research:  Agencies 
Need  to  Strengthen  Efforts  to  Improve  the  Completeness,  Consistency,  and  Accuracy  of 
Awards  Data  (GAO-07-38). 

The  National  Institutes  of  Health  supports  the  recommendation  that  the  Small  Business 
Administration  (SBA)  and  Small  Business  Innovation  Research  participating  agencies 
work  together  to  strengthen  efforts  to  ensure  that  the  data  collected  for  SBA’s  Tech-Net 
database  are  complete,  consistent,  and  accurate.  We  recognize  the  importance  of  the 
collection  of  program  results  and  look  forward  to  working  collaboratively  with  the  SBA 
to  improve  the  quality  of  the  data  in  the  Tech-Net  database. 
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GAO’S  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  pohcies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  rehability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday,  GAO  posts 
newly  released  reports,  testimony,  and  correspondence  on  its  Web  site.  To 
have  GAO  e-mail  you  a  list  of  newly  posted  products  every  afternoon,  go 
to  www.gao.gov  and  select  “Subscribe  to  Updates.” 

Order  by  Mail  or  Phone 

The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each. 

A  check  or  money  order  should  be  made  out  to  the  Superintendent  of 
Documents.  GAO  also  accepts  VISA  and  Mastercard.  Orders  for  100  or 
more  copies  mailed  to  a  single  address  are  discounted  25  percent.  Orders 
should  be  sent  to: 

U.S.  Government  Accountability  Office 

441  G  Street  NW,  Room  LM 

Washington,  D.C.  20548 

To  order  by  Phone:  Voice:  (202)  512-6000 

TDD:  (202)  512-2537 

Fax:  (202)  512-6061 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 
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Washington,  D.C.  20548 
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